Papers with language modeling ability
Towards Understanding Task-agnostic Debiasing Through the Lenses of Intrinsic Bias and Forgetfulness (2024.findings-acl)
Copied to clipboard
Guangliang Liu, Milad Afshari, Xitong Zhang, Zhiyu Xue, Avrajit Ghosh, Bidhan Bashyal, Rongrong Wang, Kristen Johnson
| Challenge: | Debiasing Pretrained Language Models (PLMs) are task-agnostic and can be generalizable, but its impact on language modeling ability and the risk of relearning social biases remain as the two most significant challenges. |
| Approach: | They propose a framework which can Propagate Socially-fair Debiasing to Downstream Fine-tuning to alleviate the forgetting issue of PLMs by regularizing debiased attention heads based on the PLM’s bias levels from stages of pretraining and debiase. |
| Outcome: | The proposed framework can Propagate Socially-fair Debiasing to Downstream Fine-tuning, indicating that the ineffectiveness of debiase can be alleviated by overcoming the forgetting issue through regularizing successfully debiased attention heads based on the PLMs’ bias levels from stages of pretraining and debiases. |
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on. |
| Approach: | They propose to use Counterfactual Data Augmentation, Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia as bias mitigation techniques to quantify their effectiveness. |
| Outcome: | The proposed techniques are Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia. |